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The starting point for innovation in higher education must include assessment, 
evaluation, and description of the present situation. A national study of what 
constitutes the core values of American academic life today should be undertaken. In 
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The Quality of the educational experience rather than the amount of specific 
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The CENTER FOR THE STUDY OF EVALUATION OF INSTRUCTIONAL 
PROGRAMS is engaged in research that will yield new ideas 
and new tools capable of analyzing and evaluating instruc- 
tion. Staff members are creating new ways to evaluate con- 
tent of curricula, methods of teaching and the multiple 
effects of both on students. The CENTER is unique because 
of its access to Southern California’s elementary, second- 
ary and higher schools of diverse socio-economic levels 
and cultural backgrounds. 



COMMENTS ON PROFESSOR TROWS PAPER ENTITLED "METHODOLOGICAL 
PROBLEMS IN THE EVALUATION OF INNOVATION" 

David Nasatir 

The overwhelming sense that I get from this Conference, and in 
part from Professor Trow's paper, is the punch line of the joke that 
ends up, "You can't get there from here." In the case of evaluation 
I am not at all sure where is "there" that I want to get to, and I 
am certainly not sure where is the "here" where we are. I do not 
know what the starting state we have at hand is; and if I cannot ade- 
quately describe the context in which I am innovating, I have serious 
problems about exporting that innovation in some other context. 
Professor Trow's remarks, although cast in the collegiate vein, are 
in fact very applicable to the problems of introducing and promoting 
innovation in general. Value innovation in the educational enterprise 
may take place at all levels; the problems associated with the politics 
of innovation are not peculiar to the university. It is necessary to 
consider explicitly the latent functions of the routines of everyday 
classroom life. 

The need for assessing, evaluating and describing the present 
situation in which an experiment will be perfoxmed, is a pressing one. 
It points out very quickly the inappropriateness of the experimental 
model of research for what is in fact a descriptive task. It is very 
difficult to make prescriptions for innovation when you cannot ade- 
quately define all the relevant conditions. 



In addition to describing the starting point for innovation we 
should- -as Professor Trow suggested- -also study the side effects. 

This brings us very quickly to the question of criterion variables, 
for one man's criterion variables are another man's side effects. An 
increase in reading scores produced by an experimental technique may 
be of less importance in determining adoption than problems of class- 
room behavior generated by adoption of the experimental technique. 

A second point raised by Professor Trow with ramifications 
for the evaluation of educational innovation in the curriculum area 
has to do with the time perspective. It often appears that educa- 
tional planners take for granted that the future is quite nebulous, 
that we do not really know what the world is going to be like, that 
we are very unsure of the kinds of skills that are going to be 
needed in the future and consequently become somewhat immobilized. 

How far away is the future? What is the time perspective that must 
be brought to bear? If we presume that future society is going to 
be much like today, I think Martin Trow's comments, while delightful, 
reveal a personal bias which I personally would not share. It is 
a strong assumption that the core values of the faculty and the stu- 
dents engaged in education, including higher education in America 
today, are those of providing and acquiring a liberal education. 

This is an empirical question, and it is my feeling that the 
overwhelming values are not liberal but vocational. While I share 



the valuation of liberal education, I do not think that is truly 
widespread at the moment. Someone should certainly do a study of 
what constitute the core values of American academic life today. 

Such a study should be national in scope and include students as 
well as faculty and administrators. It might not be a bad idea to 
include legislators and taxpayers as well. 

I doubt that many people experience the kind of liberal educa- 
tion that Professor Trow is talking about. Those who do make up 

the liberal education establishment, and those who control the 
possibility of change actually appear relatively content. They 
are not tempted by problems of assessment. I will return to this 
point, because I think it is suggestive of the importance o c broaden- 
ing the unit being studied in order to include not just the student, 
but all of the participants in the educational endeavor. You will 
recall Professor Trow suggesting that many of the innovations that we 
see are innovations which amuse the innovators and, I suspect, many 
of us who are engaged in education. Some who are engaged in educa- 
tional innovation do so only because it is amusing, because it is 
interesting; were that possibility denied us, we would turn our 
attention to more diverting things. 

Recently in a policy decision dealing with retention of some 
computer hardware at a research center at Berkeley, the issue was 
phrased, "If we don’t have the toys, we can't keep the programmers.” 



If we do not have the toys with which innovators can amuse them- 
selves, they cannot be kept around. 

If we now make the opposite assumption that social patterns 

of the future are not going to be as they are today, then Professor 
Trow's concerns are much more germane. We already see a tremendous 
expansion of formal education toward a point where it becomes co- 
terminous with life itself. This fact is evident when we look at 
the life style of graduate students at many leading universities. 
Wives and children of the students are immersed in the educational 
context; they live on the campus or in its environs; their social, 
economic, religious, political, military, and vocational experiences 
are cast in terms of the educational setting. And these conditions 
may be continuous over periods of three, five, eight, or even fifteen 

or more years at a time. 

It is not at all clear that there is a distinction between 
educational life and "real" life. As Professor Husek said to me 
one day vjhen we were bicycling along the coast toward Malibu, I 
don't know what you're in training for, but this is the event so 
far as I’m concerned." Students are not in training. This is the 

event . 

Similarly, .because of a concern of social scientists and many 
others , we see the reaching out of formal education to envelop 
the young child and to provide in the educational setting almost 



a total definition of his life. Even at present mn~e than a third 
of a person’s life --this is for the bulk of the population of the 
United States --is spent in education. Educational enterprise is 
the setting for life experiences, and I suspect this will be true 
increasingly, not only in the numbers of people involved but in 
the scope of their involvement . 

Now it may be that organizational differentiation in the 
future will remove some of the overwhelmingly vocational character 
of contemporary education. Many of today’s problems may be solved 
by a development of highly specialized educational organizations . As 
the rate of technological change increases, the length of time that 
a given skill is useful will diminish. Education, even for the 
most vocational, will never be finished, and we may expect schools 
to be reorganized so that dropping out- -and dropping in- -is the 
normal rather than the deviant pattern of attendance. Even the 
near future is sufficiently vague, the probability of rapid tech- 
nological and social changes sufficiently great, and the variety 
of alternative ' social arrangements so large that those who wish 
to emphasize the transmission of specific, readily measurable know- 
ledge might wish to hesitate a little bit and consider the merits of 
what they are doing. If we were to view the future as something differ- 
ent from a continuation of present arrangements, we might redefine the 
problem of evaluation of instruction to look, instead, at the side 
effects. We would want to evaluate the quality of the educational 



experience itself rather than the amount of specific information trans- 
mitted. We would want to look at the delight engendered by the parti- 
cipants in the process of learning rather than in what is learned. If 
education is going to become an all -pervasive element of everyday life 
and if we are going to be the technocrats who manipulate it, we should 
concern ourselves with the quality of the educational experience itself 
in humane terms. 

Who are the participants in this endeavor? Professor Trow has 
pointed out that it is folly even for the most narrow-minded of inno- 
vators in the technology of education to confine their considerations 
only to those things affecting the students. It is folly because the 
real world does not operate that way. Whatever innovations are to 
be proposed must be considered in light of their impact on all par- 
ticipants in the enterprise. This consideration determines, in 
large measure, the likelihood of adoption in the real situation; it 
also establishes the range of parameters to be considered in evalua- 
ting the worth of the innovation. It is necessary for educational 
researchers to make manifest many of the functions which the school 
performs. Often, these are in conflict with basic values school 
administrators hold about the ideal nature of the educational 
enterprise. It becomes necessary, for example, to make manifest 
the amusement value of innovation for those involved and to include 
these considerations in the cost-benefit analysis repertoire Professor 



contextual along with the experimental variables. The problem is 
that we do not have a well developed theory which allows us to go 
very far in the quantitative specification of types of contexts. 
With this shortcoming we unfortunately expend our scarce resources 
doing that which is somewhat easier to do, simply because we have 
methods developed for it, not because the proposed innovations lead 
necessarily to a more desirable end. What must be done, then, is 
to begin work on the assessment of instructional programs in their 
totality rather than focusing upon the minor aspects of them that 
correspond to existing curricular divisions. 



